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PROGRAM EVALUATION: OVERVIEW 



Evaluation is procedure that is invariably prescribed for any kind of 
educational programs, including college-level and faculty development 
courses. Even so, evaluation is seldom done in a formal, extensive manner 
for several reasons; formal, extensive evaluations can be time-consuming 
and, for busy faculty, often without reward. Further, formal evaluations can 
be saddled with methodological and procedural confusion. 

One source of confusion arises from whether the focus of the proposed 
evaluation is formative or summative. Formative evaluations, in the ideal, 
provide information about how a program is operating as it is being carried 
out. Such information reported in a timely manner allows for adjustments 
and improvements in the program as it is being delivered. Summative 
evaluations, on the other, provide information about what effects, if any, 
were accomplished by the program after it has been completed. Each focal 
procedure has its distinctive information needs and operational schedule. 

Another source of potential confiision derives from procedural uncertainty. 
That is, how should a formal evaluation actually be carried out? For 
example, practitioners have been provided four classic approaches to 
evaluation (Clardy, 1997): 

A. Experimental. 

This approach applies the traditional experimental design to test the extent to 
which an educational program made a significant difference in some 
measured outcome, like learning, student interest, application, and so on. 

The “hypothesis” is that the program will make a significant difference in 
the dependent outcomes of interest. 

B. Effects-based (goal-free). 

Here, the evaluator enters the evaluation as a blank slate, intentionally 
oblivious to emy stated goals for or planned outcomes from a program. 
Instead, the evaluator will look for any and all effects (good, bad or 
unknown) that resulted from the program. 
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C. Stakeholder driven. 



In this approach, the evaluator first identifies the different groups who have 
with an interest in a program, like students, parents, administrators, other 
teachers, and so on. The evaluator learns what each group of “stakeholders” 
expects from the program, and then looks to see to what extent each of those 
expectations have been met. 

D. Goals-based. 



This is the classic - and most obvious - approach to program evaluation. It 
is based on examining the extent to which the specific, stated goals or 
objectives of a program have been achieved. In a sense, this classic 
approach is a benchmark procedure against which to compare other types of 
evaluation and assessment procedures. 

A precis of the goals-based approach to program evaluation is provided on 
the following page. 



This report describes a relatively new approach to program evaluation - the 
so-called “learning history” approach, presents preliminary results of a pilot 
test of this method, and provides a preliminary account of how this new 
approach will be compared to the classic goals-based approach to evaluation. 
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THE GOALS-BASED APPROACH TO EVALUATION 







Type of evaluation 


Summative 


Focal question 


To what extent were stated and explicit goals met? 


Principal data 
collected 


Indicators of achievement on stated and explicit 
goals 


Requirements for 
doing this evaluation 


Program goals and objectives must be identified 


Procedure (assumes 
requirements are in 
place) 


1) Operationalize the goals and objectives into 
measurable indicators of performance. 

2) For each indicator, establish a standard to 
distinguish between success and failure. 

3) Collect information about performance on these 
indicators. 

4) Compare performance to standards; How well 
were program goals achieved? 


Timing of data 
collection 


Generally conducted after a program has been 
completed 


Advantages of this 
approach 


Strong accounting for how well program worked in 
terms of program expectancies and desired outcomes 


Disadvantages 


Neglects other effects and conducted after the fact 



THE LEARNING HISTORY APPROACH 



Background 



Early in the 1990’s, this method was developed by Roth and Kleiner at MIT. 
It derives from the discipline of Organization Development and emerging 
interests in organizational learning. This technique was originally created as 
a means for generating organizational learning about change programs in 
work organizations. See the list of suggested readings included. 

Note: in their own way, that’s what formal evaluations should do: provide a 
means for learning about a change program (an educational course). 
Questions: 

1. can this technique be used as a method of program evaluation? 

2. how does this method compare to the classic goal-based approach to 
evaluation? 



What is a Learning History? 

A learning history is defined in terms of its process and its reporting format. 
As a process, the evaluator solicits information from a number of 
participants about their experiences over the course of the change program. 
The primary purpose of the process is to spur “reflective conversations” 
about the nature of the change process from a variety of viewpoints. In turn, 
this document is made widely available to members of the organization. 
Finally, members of the organization meet for a collaborative review of this 
document and its contents; this review generates ideas and conclusions that 
promotes organizational learning. 
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Learning History Format 



As a report, a learning history becomes a document like a scrapbook of 
memories and experiences of a change process. It is presented in a specific 
format: 

• The major segments, stages or episodes of the process are identified, 
and they become the “chapters” of the report. 

• Each chapter is headed with a provocative title. 

• A brief narrative description of each segment, in terms of timing, 
participants, events and activities, conditions, etc., is included. 

• A two-column format is used to present the data: 

1) Representative or significant quotes from the participants are 
included in the right column; participants are identified by job title 
only. 

2) The left column is used for interpretations, questions, themes or 
commentary 

The document may also contain other information, too, like memos, 
announcements, or other documents. 

A precis for the Learning History approach is also provided. An illustration 
of data-collection forms that could be used to create a learning history for 
this Lilly Conference is included as Appendix A. 
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THE LEARNING HISTORY APPROACH TO 

EVALUATION 



Type of evaluation 


Summative and Formative 


Focal question 


What happened, whether planned or not? What can 
be learned from that experience? 


Principal data 
collected 


What participants observed and thought about their 
experiences in the program 


Requirements for 
doing this evaluation 


Access to a large number of participants and ability 
to obtain commentary from those participants 


Procedure (assumes 
requirements are in 
place) 


1 . Planning the boundaries and scope of the project 

2. “Reflective” or depth interviews with participants 

• team of insider participants and outside 
researchers 

• diverse and large sample of people 

• other data can also be gathered 

3. Common and recurring themes are sought 

4. Writing Draft 1 using the prescribed format 

5. Validation of data through participant review 

6. Group analysis of the document in workshops, 
where participants review the history to look for 
ideas for future practices based on what they learned 
from experience 


Timing of data 
collection 


Best if data is collected while the change program is 
occurring (better contemporaneous reporting by 
participants) 


Advantages 


“Learning histories seem particularly effective at 
raising issues that people would like to talk about but 
have not had the courage to discuss openly” 


Disadvantages 


Labor and time intensive as proposed 



ADAPTING THE LEARNING HISTORY APPROACH 
TO COURSE EVALUATION 



I adapted this approach for use in a web-supported graduate course I taught 
at Towson University in order to pilot test this approach as a method course 
evaluation. About this same time, I was begiiming a study that would 
compare this approach to a goals-based evaluation method of a faculty 
development program. 



Pilot Test: A Learning History of the Change in the Workplace Course 

In the Fall Semester, 2000, 1 used a Learning History approach as part of a 
required graduate course I taught in the Towson University Human 
Resources Development program. The course is called Change in the 
Workplace. One primary instructional goal for this course was to teach 
methods for producing organizational change. In that context, one topic that 
was taught was that of the “learning organization.” The Learning History 
approach was used as an instructional demonstration of this topic. 

This course was taught with web-support. This meant that even though 
classes were held once weekly during the semester and students were 
expected to attend all classes, they were also expected to make extensive use 
of the web-based instructional system designed for the course (Blackboard 
5). In particular, this meant that students were required to make regular 
comments on what they were learning and their reactions to the instructional 
experience. 

One main point of adaptation of the learning history method in this case was 
that in-person in-depth interviews were replaced by the on-line messages 
and reports provided directly by students. Thus, data from all student 
participants was collected on-line and was provided directly by the students 
under a loosely structured schedule. Each class was its own chapter. 

In writing the learning history, I made contemporaneous entries to the 
history after each class. Every few weeks, the updated copy of the learning 
history of the course was posted to the course web site where all students in 
the class could read it. Samples of this initial history follow. 
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Samples of the Learning History, Change in the Workplace course, Fall Semester, 2000. 
Dr. Alan Clardy, Instructor 



CLASS 1. BREAKING THE ICE, SETTING THE TABLE 

Dr. Clardy began the first class promptly at 7 p.m. After reading the names on the 
registration list, he spent about 30 minutes going over the contents of his web page and 
the syllabus for the course. This was followed by introductions of himself and then each 
class member. After the break, the class worked in small groups looking at the case of 
Redwood National Bank. The class adjourned about 9:15 without completing the final 
section of the case study. Dr. Clardy posed a question to help frame the purpose and 
direction of the course: how can you efficiently and effectively implement changes like 
those seen in the case study? 



Are these the best goals for 
opening a class? How well 
are these goals met by the 
procedures used? 

This can be an example of 
the difference between 
espoused theories and 
theories in use. How well 
was the articulated intention 
really carried out? Was 
there another agenda which 
really guided what happened 
and of which the instructor 
might not have been aware? 



Learning goals seem to be 
of several types: 

• Complete the course 
(with an A) in order 
to graduate 



Instructor: I always try to set a tone and climate for the 
course at the start in several ways, by being clear about 
expectations, by involving students personally (through 
introductions), and by engaging them in a fundamental 
problem which the course will be examining. The group 
project is intended as much to set a norm of discussion 
and participation as it is to process content. I was very 
mindful of trying to work in the Courseinfo resource 
material, particularly during the syllabus overview. The 
case study still needs to be refined: it takes too long to get 
through it, and we did not even get to the third part. Turns 
out students seemed to be interested in how LW and I 
reacted to the defensive denials of the executive team in 
the debriefing meeting we conducted at “Redwood.” 

Student 1 : My learning goals and objectives are two-fold: 
first, to do the best work that I am capable of while completing 
the course assignments (since this course is required as part of 
my degree) and second, to learn how to effectively incorporate 
change into any organization, since change is inevitable. 

Student 2: My learning goal is to gain some REAL tools that 
can be used in facilitating organizational change. Theory can be 
useful, but I am looking for tools to use. I would like to develop 
work that I can use as part of my portfolio. 



• Learn about how to 
manage 
organizational 
change 



Student 5: 1 am still deciding what my learning goals are for 
this course, partly because I am still figuring out what the 
course is about. The syllabus and information on Courseinfo 
have given me some clues that the course involves more 
introspection then I suspected. 
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• Not sure what their 
goals are 



Student 12; There are several things I would like to get out of 
this course. I would like to become proficient at the Blackboard, 
earn an "A", and learn to implement procedures of change in 
the workplace from the practice end of the continuum rather 
than the theory end. 



Class 8. PARADOX: HOW CAN WE ALL KNOW SOMETHING THAT NO ONE OF 
US KNOWS? OR, WHAT’S THE SOUND OF ONE HAND CLAPPING? 

Dr. Clardy reminded the class to complete the peer assessment within the next week or 
so. The first 40 minutes of this class were spent reviewing two key concepts left over 
from last week: employee attitudes about change and the contingent approach to change 
management. The remainder of the class was spent doing the Organizational Learning 
simulation of Letz Parti. There was a rapid debriefing on the points. Finally, the first 
graded exam papers were returned. 



Change in classroom 


Instructor; I was mindful of starting on time (even 


management in response to 


though only about 2/3 ’s of the class was present) and did 


survey feedback 


not feel badly about keeping the class until almost 9:30. 
This was in response to a survey feedback comment. 


Learning about 


I’m ambivalent about the Letz Parti simulation, and am 


organizational learning 


wondering whether I should reconsider using it. Is it the 
best way to convey the idea of a Learning Organization? 


Ideas for next time: 


My observations. 


There were favorable 


• the production simulation is good, I think; 


student comments about the 


• where it may breaks down is in the use of 


production phase 


environmental signals and learning both how the 
world is changing and how the organization needs 


Provide an advance written 


to change itself; this may be more of an 


explanation of what the 


explanation problem than a structural problem with 


simulation is 


the simulation itself 

• Td like to do more about the learning piece, both 


Provide element on 


in terms of organizational learning disabilities and 


organizational learning 


in organizational learning procedures 


disabilities 


• I didn’t link the discussion of knowledge 

management with either the simulation or with the 


Deal with linkages to the 


other activities we are doing in the course (through 


course’s knowledge 


Course Info Discussion Board and this learning 


management system 


history) to create a knowledge management system 
of the course 



Class 10. CHAOS! 



Tonight, we followed a poster presentation format to allow students to present their 
professional practice papers reporting on a change model they studied. We used three 
rounds of about 5 each round. There seemed to be reasonably good levels of 
participation. The following organizes this discussion around these three issues. 

1) The Poster Session Format 



Confusion by students over 
what a “poster session” is 


Instructor. CHAOS was the way some people seemed to 
react to this, in spite of at least two separate sets of 
instructions about what will happen. Upon reflection, it is 
clear that most have not done anything like this. They 
came in, as a general rule, with poor to no visual aides. 


Suggestions for organizing 
the classroom presentations: 
provide guidelines for how 
much time to spend at each 
station 


Student; 1 thought this was an awfully good idea and 
generated good class participation. Many of the students 
seemed to enjoy presenting their findings on different subject 
areas to others. 1 liked the idea of having 4 or 5 stations where 
students could present - one suggestion for next time - set it up 
so those students listening need to take 5 minutes of so to stop 
at each station. What 1 observed was that some stations were 
very busy, while others were not. But overall, a very nice idea. 



3) Selecting a Model of Change 



Providing a list of approved 
references was not sufficient 


Instructor . In spite of the book list, there were a few poor 
choices used. One was a few pages from a text used in 
another class, a second was a pseudo-philosophical 
treatise with little arguable relevance to the course, a third 
was a complete book used in another class. Further, many 
used books that simply describe an end-state without any 
model for how to get to that end state. 


The quotation is edited 
somewhat [the changes 
made are in brackets] 


Student; I have a concern that I have to "get off my chest" 
regarding the last project. I felt insulted in the presentation on 
[a leadership] book because I had the understanding that we 
(the grad students) had to pursue more "proven" or researched 


Other students pay attention 
to the quality of the work 
provided by their classmates 


methods of change management—not a self-help book. I know 
that you have final discretion to approve whatever book you 
deem appropriate— but I felt that [that student’s] selection was 
both degrading and diminishing to the level of studies that we 
have pursued in the course so far. To make matters worse, [this 
student who used this text] was bragging that he was reading 



the book for another class. . . . There were other students that 
researched and presented materials on a graduate level, 
presented arguments about why the author's theories would 
work or not. To me, THAT was the purpose of the learning 
exercise. I am proud of the work that most of the class has 
done. I take the opportunity of grad school as a serious 
commitment“to the instructors and to myself When people 
"slide by" with such mediocre effort, I feel angry. 



Recommendations on the Professional Practice Review Paper Assignment 



What seems to be indicated 


Changes to the syllabus; 




* You must use a visual aide and it should be a poster- 
sized or better 

* Do I require them to provide a handout if their paper 
will be downloadable? Maybe not. 

* Require that all students email me with their selection, 
by the 4**“ week of the semester. I have to approve it, and 
if I don’t, then I may reject their paper. 

* Spell out more clearly in the syllabus and the lectme the 
difference between a volume that specifies some end state 
and one that provides guidance about how to get there. 
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Mv Observations about the pilot test 

A. In writing the history, I had to be conscious of wearing two hats: 

1) That of a participant in the program with my own experiences as 
instructor and a member of the course/classroom system 

2) That of the “outside” reporter and interpreter of the process 

In general, I think I could effectively separate the two roles and perform the 
authoring duties of brief descriptions of what happened in each class, 
including a range of student and instructor comments, and adding themes 
and commentary. 



B. I was now making visible for public inspection my internal dialogue 
about my experiences in lesKling the class. This included any doubts and 
fiustrations I was having, and issues I would have kept to myself were now 
expressed in this document. This was a new experience for me, and I found 
that by writing about my experiences in teaching it was both a catharsis as 
well as an inflammation. Sharing my concerns was a relief in some ways. 
However, the act of putting these concerns in writing also tended to facilitate 
a concentration on certain issues, and I would began to dwell on certain 
problems which were now “public”. 



C. Few students read the report; only one student responded to me about it. 
By the time I could finish the full report for the course, it was the semester 
end and students not available to for group meeting 



D. I used the report extensively in improving how I taught the course. Even 
though this was the fifth time I taught it, I identified more than 15 significant 
changes to make to the course based on a review of the final report. 



E. The Learning History approach can be adapted and used as a method for 
educational program evaluation. 
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COMPARING GOAL-BASED AND 
LEARNING fflSTORY APPROACHES: 
A PRELIMINARY REPORT 



Background 

Given that the learning history approach to program evaluation is feasible, 
how does it compare to the classic goals-based method of program 
evaluation? The relative cost and effectiveness of both approaches can be 
assessed. To do this, I am conducting a study comparing these two 
approaches in an evaluation of a faculty training program at mid-Atlantic 
university. The program was held over the summer, 2000 for a dozen 
faculty representing all the major colleges at this university. The program 
taught the participants how to use a web-based instructional system recently 
installed in the campus. In exchange for their paid participation in this 
program, faculty members agreed to design and use web-support in a course 
they would teach in the following year. 



Research Comparison Study: A Preliminary Report 

This study covered here will focus on examining two questions: 

1. What is the relative cost-effectiveness of the learning history approach 
compared to the goal-beised method of program evaluation? 

2. Instead of in-depth interviewing, what are the implications of using of an 
email-based, survey approach for data-collection? 

The study is still on-going. To date, I have interviewed program principals 
to identify the background to and goals for the program. These interviews 
also yielded information for dividing the program into seven segments, 
providing a framework for collecting the data. Using this framework, a 
survey form was prepared and sent by email to all parties involved in the 
program, including faculty, program administrators, trainers and student 
aides. The survey form asked the seven questions for each segment; the 
basic format is shown on p. 15. Respondents typed in their answers and 
returned the survey on-line. A sample of the resulting learning history is 
included on pages 16-18. 

o 
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PLEASE COMPLETE AND RETURN TO ALAN CLARDY BY JANUARY 17, 2001. 
Phase 1. Off to the Races: Planning for the <PROGRAM> 

Approximate dates of the episode : February 1, 2000 to March 30, 2000 



Synopsis of what happened : After the Fall and January, 2000 experience with the 
Provost’s initiative, the <PROJECT TEAM> began a series of discussions in February to 
plan for a <PROGRAM>. These discussions included the Provost. During this time, 
funding from <A CABLE COMPANY> was also obtained. Certain program 
parameters, learned from the fall initiative, were included in the design of this program, 
including: competitive bidding among faculty to participate, providing financial 
incentives to participate, scheduling the program over the summer when the faculty 
would have more free time, “upping” the courses targeted (from lower-level to upper 
level undergraduate and graduate course offerings), and reducing the amount of on-line 
presence (to about 25%) required from the program. This phase of the program 
culminated in a two-page Aimouncement that was released to the faculty in the early 
spring, 2000. 



Please recall your involvement in the episode identified above. Feel free to refer to any 
notes, manuals, calendars, or other memory aides that you might have. Please answer the 
following questions: 

1. Did you have any specific thoughts or feelings about the <PROGRAM> or your 
involvement in it when this episode began? If YES, please describe. 

2. What were the most significant or important thing(s) that happened during this 
episode, as you recall? 

3. How would you describe your involvement or participation in this episode? 

4. What problems did you experience or issues did you see in this period? 

5. Please add any other comments or observations you have of this period and the 
activities that took place in it. 

6. What was your assessment of the <PROGRAM> at the end of this period? 

7. Do you have any recommendations or suggestions for how to correct or avoid 
problems, or to make the Institute more effective in the future? 
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Episode 3. Let the Games Begin: Kick Off 

Approximate dates of the episode : May 3 1 (plus and minus a few days) 

Synopsis of what happened: On May 3 1 , the first kick-off meeting of the participants 
selected for the program was held in a computer lab room in University Library. The 
two-hour meeting included a round of introductions, plus an overview of the program. 
Participants received a book about teaching on-line. 



Beginning the process, faculty 
entered with various issues 
and concerns; 


Assistant Professor B; I remember I was very nervous 
when I found out that almost everyone else was more 
advanced than me in terms of the courseinfo software. 


Uncertainty about one’s skills 
seemed to produce a reticence 
and reserve 


Instructor. I tried to stay low-key, as I knew others had 
wider experience in the field. <From this meeting,> it 
was apparent from this get-together that planning and 
solid university administrative support were built into 
this summer program. 


Questions about what 
participants were supposed to 
do in the Institute 


Assistant Professor C: I didn’t attend the meeting 
<but> after getting information from a coworker, I was 
concerned about all of the “requirements” of 
participation that was not disclosed up front during the 
application process. 




I liked the book a lot. 




Assistant Professor D: While I was interested in the 
content, I was anxious about who would be there, what I 
might be expected to do during that session, and so on. 
Some of that anxiety derived from a status change a 
participant in the program had experienced and my 
reactions to that. In short, I was stiff and unresponsive 
during this meeting; this attitude prevailed during the 
remainder of the Teaching Institute, for basically the 
same reasons. There was also some uncertainty about 
my computer skills. 


Initial introductions were 
insufficient to prepare for best 
participation 


I thought the most significant thing about this episode 
was what did NOT happen; for a group that would be 
working together over the summer, the introduction 
phase was short and did not use any ice-breaker type of 
activity. While I knew some more details about the 
others, there was no chance to get to know them 
informally or begin to establish any social ties. 
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Incomplete information 
provided in advance? 



Questions about attendance 
arose from the start 



A goal of the Institute was to 
create learning and mutual 
support cohort 

But 

Early emphasis on teaching 
strategy was too soon 



My involvement or participation in this episode was 
attentive but perfunctory and restrained. I asked to be 
excused from the initial training session that covered the 
basics of using the on-line instructional system, which I 
felt I already knew. I had a feeling this request was 
reluctantly granted. 

We received the summer schedule for the program for 
what I thought was for the first time. I believe that it 
may have been distributed as part of the initial 
announcement. I remembering feeling surprised in 
seeing it for what I thought was the first time. 

I would recommend spending more time in building 
personal relationships between participants through ice- 
breaking and/or cocktail-type social gathering. 

Actually, I’d recommend doing both. 

Program Manager; It was really important that 
everyone attend the kick-off session. Yet some were 
unable to attend and that set a tone for the remainder of 
the Teaching Institute. The Administrator was pretty 
upset about the attendance problems. 

One of the goals of the Teaching Institute was to have 
faculty work in cohorts throughout the Teaching 
Institute to share ideas and learn from each other. There 
was such a wealth of teaching experience in the 
Teaching Institute group, we wanted to harness that 
experience to the benefit of the participants. We tried to 
group participants by teaching strategy beginning with 
the kick-off session, but it didn’t work as well as I 
hoped. In retrospect, I think it was too early to decide 
what strategies participants wanted to use. In several 
cases, participants selected teaching strategies at the 
beginning of the Teaching Institute and after rethinking 
their courses for partial online delivery, they switched 
to using other teaching strategies. In fiiture institutes I 
would form cohorts later. 

I hoped the book by Heidi Schweizer would be well 
received. 

Otherwise I think the kick-off session went well. 



Recognition issues among 
staff 



Possible problem; unclear 
understanding of what the 
faculty were wanting to gain 
from the Institute 

This condition created by the 
rapid deadlines faced 



Recommendation; meet with 
faculty personally prior to start 
to identify what they want to 
learn 



Support Staffer A: As a member of the ‘team’- 1 felt 
somewhat like an afterthought. <The other staffers> 
worked on the workshop planning nearly independently 
and clearly were the ‘top of the bill’ during the training 
sessions. I was just learning the on-line instructional 
system myself- so I had more in common with the 
faculty particip)ants. But the constant focus on the other 
support staff got old after a while. 

As a kick-off- 1 though it was ok- I’m not sure how you 
all felt. 

Program Administrator . I'm not sure everyone was 
on the same page at that first meeting. I thought there 
was an opportunity for everyone to get a better sense of 
what was going to take place in the summer. It was also 
a way for the faculty to get to know one another. 

Because we were ending the Provost's Web Course 
Initiative at the end of April and starting the Teaching 
Institute at the beginning of May, we did not have to the 
time to talk to each of the faculty in the Teaching 
Institute to have more information on what each of them 
expected to get out of the Teaching Institute. This 
would have been valuable information BEFORE the 
kick-off meeting to make that meeting more meaningftil 
to the faculty as they started the Teaching Institute 
activities. I feel we were already a bit off target in 
giving the faculty what they wanted and needed. 

<I would recommend> at least one substantive lonl 
meeting with each faculty member after the person is 
accepted into the program and before the kick off 
meeting. Then a very directed meeting of the team to 
hear what the faculty members will be doing and what 
they wish to out of the Institute so that the kick off 
meeting draws the faculty into the process and helps 
them to hit the ground running. 



Non-responses to questions 
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Preliminary findings and implications 

1. Program goals were not clearly identified at the start of the program. For 
various reasons, including last-minute fimding and the need for rapid 
delivery plus changing university priorities, the goals for this program were 
not explicit at the start of the program. Rather, goals were implicit and 
informal at the start of the program and only became explicit and specific 
through discussions after the program had ^eady been completed. 

2. Some of the identified goals were conflicting, depending on the 
stakeholder group consulted 

3. E-mail responses to the questions were often short but did reveal thought 
and consideration. 

4. There was a 50% response rate from faculty; 100% from the five 
administrative principals and trainers involved. 

5. It took approximately 7 hours to put together and edit the email entries 
into a learning history document that included direct quotes fi'om all 
participants. This data was all collected in a month’s period of time. 

If in-person depth interviews were used and assuming an hour an interview, 
it would have taken 12 hours just to collect the raw data. That information 
would then have to be screened and transcribed (assume another hour). In 
short, it probably would have taken two hours per interviewee for a total of 
24 hours. Further, there would be time required to schedule interviews. 

Plus the inevitable delays and cancellations would likely mean that it would 
have taken longer to get the data. It is unknown how much more 
information would have been revealed fi'om the depth interviews, but it 
presumably would have been more. 



The next steps for the research are to complete the goals-based assessment. 
The learriBLig history should be distributed for comment. Finally, using a 
sample of administrators and participants, the two evaluation reports would 
be compared in tenns of value, time, interest. 

The result of this comparative study will be reported in a study intended for 
publication. 

o 
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Appendix A. Illustration of data-collection forms that could be used to 
create a learning history for the Lilly Conference. 

I. Segments of the Program. Based on a brainstorming activity among 
participants in this presentation, several major stages to the Conference can 
be identified, along with a provocative heading and a brief statement of the 
major events that happened during that segment. 



STAGES, EPISODES 


HEADING 


MAJOR DESCRIPTORS 





















n. Preparing a concise chapter report. This template shows how the information 
could be applied to a Learning History format. 

Stage : 

Description 
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